Alternate viewpoint image enhancement

ABSTRACT

In one embodiment, panoramic images, images bubbles, or any two-dimensional views of three-dimensional subject matter are enhanced with one or more alternate viewpoints. A controller receives data indicative of a point on the two-dimensional perspective and accesses a three-dimensional location based on the point. The controller selects an image bubble based on the three-dimensional location. The three-dimensional location may be determined according to a depth map corresponding to the point. A portion of the image bubble is extracted and incorporated into the two-dimensional perspective. The resulting image may be a seamless enhanced resolution image or include a picture-in-picture enhanced resolution window including subject matter surrounding the selected point.

FIELD

The following disclosure relates to the enhancement of images innavigation-related and/or mapping-related applications.

BACKGROUND

Conventional street view involves the manipulation of image bubbles.Image bubbles are panoramic images collected with respect to a singleperspective. The image bubbles may extend nearly 360 degrees along oneor more axes. Zooming and panning within an image bubble provides a viewfrom a particular viewpoint but cannot provide any perspective of theview with respect to surrounding areas. To view another perspective, atypical street level viewing application must be switched to the nextimage bubble. That is, in order to view the scene further down thestreet, a user selects a direction to move to the next image bubble andwaits for the next image bubble to load. A panning operation within asame bubble may also be required, resulting in loading multiple images.These operations are tedious and require time and bandwidth to loadmultiple images and/or image bubbles.

SUMMARY

In one embodiment, panoramic images, images bubbles, or anytwo-dimensional views of three-dimensional subject matter are enhancedwith one or more alternate viewpoints. A controller receives dataindicative of a point on the two-dimensional perspective and accesses athree-dimensional location based on the point. The controller selects animage bubble based on the three-dimensional location. Thethree-dimensional location may be determined according to a depth mapcorresponding to the point. A portion of the image bubble is extractedand incorporated into the two-dimensional perspective. The resultingimage may be a seamless enhanced resolution image formed from multipleimage bubbles or an image that includes a picture-in-picture enhancedresolution inset including subject matter surrounding the selectedpoint.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention are described herein withreference to the following drawings.

FIG. 1 illustrates an exemplary navigation system.

FIG. 2A illustrates an exemplary perspective view of a geographic regionfor data collection and the associated panoramic image bubbles.

FIG. 2B illustrates exemplary panoramic image data and depth datacollected in the region of FIG. 2A.

FIG. 3 illustrates an exemplary overhead view of a geographic region andassociated panoramic image bubbles.

FIG. 4 illustrates exemplary composite image data of alternateviewpoints.

FIG. 5 illustrates an exemplary alternate view selection.

FIG. 6 illustrates an exemplary display of the alternate view selectionof FIG. 5.

FIG. 7 illustrates another exemplary alternate view selection.

FIG. 8 illustrates an exemplary overhead view of FIG. 7.

FIG. 9 illustrates an exemplary display of the alternate view selectionof FIG. 7.

FIG. 10 illustrates another exemplary display of an alternative viewselection including a see-through model.

FIG. 11 illustrates an exemplary mobile device of the navigation systemof FIG. 1.

FIG. 12 illustrates an exemplary server of the navigation system of FIG.1.

FIG. 13 illustrates an example flowchart for alternate viewpoint imageenhancement.

FIG. 14 illustrates an exemplary transition sequence for alternateviewpoint image enhancement.

FIG. 15 illustrates an exemplary transition sequence with multiplepoints.

DETAILED DESCRIPTION

The disclosed embodiments relate to methods and systems for enhancing astreet side image with another street side image. The two images arecaptured from different geographical areas and/or differentperspectives. The current image as used in a map-related ornavigation-related application is combined with the other image, whichmay have been captured down the street or at a nearby location. In oneembodiment, the images are combined to form a seamless composite image.The composite image includes portions of images captured at multiplelocations. The multiple locations are closer to the subject matter indifferent portions of the composite image. Therefore, the compositeimage may have a greater spatial resolution than any of the capturedimages. In another embodiment, the image captured down the street or ata nearby location, is used to see around or through an obstacle or toprovide a zoomed-in view of an object. The second image may be presentedin a window or a picture-in-picture arranged in the current image.

FIG. 1 illustrates an exemplary navigation system 120. The navigationsystem 120 includes a map developer system 121, a mobile device 122, aworkstation 128, and a network 127. Additional, different, or fewercomponents may be provided. For example, many mobile devices 122 and/orworkstations 128 connect with the network 127.

The developer system 121 includes a server 125 and a database 123. Theoptional workstation 128 is a general purpose computer includingprogramming specialized for the following embodiments. The workstation128 includes at least a memory, a processor, and a communicationinterface. The developer system 121 may include computer systems andnetworks of a system operator (e.g., NAVTEQ or Nokia Corp.).

The mobile device 122 is a smart phone, a mobile phone, a personaldigital assistant (“PDA”), a tablet computer, a notebook computer, apersonal navigation device (“PND”), a portable navigation device, and/orany other known or later developed portable or mobile device. The mobiledevice 122 includes one or more detectors or sensors as a positioningsystem built or embedded into or within the interior of the mobiledevice 122. Alternatively, the mobile device 122 uses communicationssignals for position determination. The mobile device 122 receiveslocation data from the positioning system.

The developer system 121, the workstation 128, and the mobile device 122are coupled with the network 127. The phrase “coupled with” is definedto mean directly connected to or indirectly connected through one ormore intermediate components. Such intermediate components may includehardware and/or software-based components.

The computing resources necessary for enhancing an image with analternative viewpoint may be divided between the server 125 and themobile device 122. In some embodiments, the server 125 performs amajority of the processing (“server-based embodiments”). In otherembodiments, the mobile device 122 or the workstation 128 performs amajority of the processing (“endpoint-based embodiments”). In addition,the processing is divided substantially evenly between the server 125and the mobile device 122 or workstation 128 (“hybrid embodiments”). Asan example, the following initial discussion focuses on server-basedembodiments.

In the server-based embodiments, an image is displayed on workstation128 or mobile device 122. The image may be part of an image bubble,which is a type of a panoramic photograph of a geographic region. Theimage may be referred to as a two-dimensional perspective.

The server 125 is configured to receive data indicative of a point onthe two-dimensional perspective. Based on the subject matter of theimage, the point is associated with a desired object. The point may be aselection made by a user at the workstation 128 or mobile device 122.Alternatively, the point may be determined internally by server 125. Forexample, the server 125 may select the point by analyzing the image todetermine where higher spatial resolution or an alternate view may bedesirable. The server 125 may also select a point based on a mapdatabase, navigation route, or destination point of interest.

The server 125 accesses data indicative of a three-dimensional locationof the selected point. The three-dimensional location may be determinedusing a depth map associated with the two-dimensional perspective. Thedepth map is an image whose pixels are made up of depth values. Thedepth values may be referenced to the location where the depth map wasgenerated, which may be referred to as the origin. The origin may be thesame location that the two-dimensional perspective image was captured.The depth values may correspond to distances from the origin to theobjects depicted in the image.

The server 125 references the point in the two-dimensional perspectiveimage in a lookup of the depth map. The server 125 receives a depthvalue from the depth map that corresponds to the point on thetwo-dimensional perspective image. Based on the depth value, the server125 calculates the three-dimensional world location of the selection.The three-dimensional location is calculated using the position andorientation of the image bubble along with the vector direction of theselected point relative to the center and the distance along the vectoras defined by the depth map. The three-dimensional location is then usedto select an appropriate image bubble. The depth map may or may not havebeen collected from the origin of an image bubble. The image bubble maybe the closest image bubble to the desired object at the point on thetwo-dimensional perspective image. The image bubble may be chosen suchthat the image has an unobstructed view of the selected point. The depthmap may also provide a surface normal at the selected point. An imagemay be selected based on which image bubble has the best straight-onview of the surface. The server 125 incorporates at least a portion ofthe image bubble into the two-dimensional perspective image. The portionof the image bubble illustrates the desired object that was previouslyselected.

The database 123 may also store image bubbles and/or two-dimensionalrenderings of geographic regions as portions of the image bubbles. Theimage bubbles may be stored in a format such as JPEG, Bitmap, Pixmap,Tiff, or another format. The database 123 may store the depth maps orpoint clouds. Each value in the depth map or point cloud may beassociated with a pixel in one or more image bubbles.

In addition, the database 123 of the navigation system 120 may be ageographic database. The geographic database 123 includes informationabout one or more geographic regions. Each road in the geographic regionis composed of one or more road segments. A road segment represents aportion of the road. Each road segment is associated with two nodes(e.g., one node represents the point at one end of the road segment andthe other node represents the point at the other end of the roadsegment). The node at either end of a road segment may correspond to alocation at which the road meets another road, i.e., an intersection, orwhere the road dead-ends. The road segment data record may include datathat indicate a speed limit or speed category (i.e., the maximumpermitted vehicular speed of travel) on the represented road segment.The road segment data record may also include data that indicate aclassification, such as a rank of a road segment that may correspond toits functional class. The road segment data may include data identifyingwhat turn restrictions exist at each of the nodes which correspond tointersections at the ends of the road portion represented by the roadsegment, the name or names by which the represented road segment isknown, the length of the road segment, the grade of the road segment,the street address ranges along the represented road segment, thepermitted direction of vehicular travel on the represented road segment,whether the represented road segment is part of a controlled access road(such as an expressway), a ramp to a controlled access road, a bridge, atunnel, a toll road, a ferry, and so on.

The geographic database 123 may be used for mapping related features ornavigation related features. The navigation-related features may includea route calculation application. End users may access a route from anorigin to a destination. The route calculation application determinesthe route for the end user to travel along the road segments to reachthe desired destination. In order to calculate a route, the routecalculation application is provided with data identifying a startinglocation (origin) and a desired destination location. In one embodiment,the starting location may be the end user's current position and thedestination may be entered by the end user. Given at least theidentification of the starting location (origin) and the desireddestination location, the route calculation application determines oneor more solution routes between the starting location and thedestination location. A solution route is formed of a series ofconnected road segments over which the end user can travel from thestarting location to the destination location. When the routecalculation application calculates a route, the application accesses thegeographic database 123 and obtains data that represent road segmentsaround and between the starting location and the destination location.The road calculation application uses the data to determine at least onevalid solution route from the starting location to the destinationlocation.

The at least one valid solution route may be displayed to the user inthe rendering of the geographic region such that addresses or points ofinterest along the route may be selected to display street side imagery.Further, an image that corresponds to the destination location, originlocation, or any points of interest along the route may be extractedfrom corresponding image bubbles and displayed to the user in therendering of the geographic region.

The route calculation application may attempt to find a solution routethat takes the least time to travel. The segment cost or travel time forthe particular represented road segment considers the type of road, suchas freeway or residential street, speed limit and distance of thesegment. In one embodiment, the route calculation application mayconsider traffic conditions to more accurately reflect actual traveltime over the connected road segments. When the route calculationapplication determines one or more solution routes comprising the seriesof connected road segments, the travel times for each of the includedconnected road segments is summed to provide an estimated route traveltime. Based on the route travel time, the route calculation applicationselects the quickest route. Once the route calculation application hasselected the route, the route calculation application provides an outputin the form of an ordered list identifying a plurality of road segmentsthat form the continuous navigable route between the origin and thedestination. In addition, the route calculation program provides anoutput of an estimated route travel time.

The map-related features may be any of the navigation-related featuresprovided to the user without reference to the current location of theuser or the device. In addition, map-related features may includedisplay and manipulation of a map of a geographic region. Themap-related features may be provided without navigation-relatedfeatures. The road segments may be tied to the image bubbles andrenderings of geographic regions.

FIG. 2A illustrates a geographic region 130 for data collection of imagebubbles 131. The image bubbles 131 include photographic image datacollected by a camera. Each image bubble may be a panoramic image 132including a graphical representation of the geographic region 130. Theimage bubbles may have center points measured in Cartesian coordinates,such as an X-coordinate, a Y-coordinate, and a Z-coordinate and anorientation defined by up to three rotation angles. Each point on theimage bubble is defined by the center point and one or more angles(e.g., roll, pitch, yaw). The camera may be mounted on a vehicle thatmoves along a street. The camera may collect an image bubble at regulartime or distance intervals (e.g., 1 m, 4 m, 10 m, 1 second, 10 seconds,100 milliseconds). The image bubbles are associated with the geographiclocation of the camera and/or vehicle at the time the image bubble iscollected.

In addition, the vehicle may include a light detection and ranging(LiDAR) device that collects and gathers data points in a point cloudusing optical remote sensing. The point cloud may be referred to as adepth map. Alternatively, the depth map may be derived from the pointcloud. For example, the data points may be matched with image bubblesbased on location. The data points include location coordinates and anon or off value. An intensity value may be included in place of the onor off value. Alternatively, the data points may include a distancevalue and two angle values to signify a location of a point in the pointcloud. The point cloud may be stored in ASCII or LiDAR exchange format.One or more lasers of the LiDAR device may be in a near infraredspectrum (such as about 700 nm to about 5000 nm or about 800 nm to about2500 nm) or another light spectrum.

FIG. 2B illustrates exemplary image data and depth data collected in theregion of FIG. 2A. FIG. 2B illustrates that some or all of the imagepixels in the image bubble 132 a have a corresponding value in the pointcloud 132 b defined by depth map data. In other words, the image bubble132 a is correlated to the point cloud 132 b and underlying depth map.

FIG. 3 illustrates an exemplary overhead view 140 of a geographic region130 of FIG. 2A. The overhead view illustrates that the image bubbles 131are spaced along the roadway 134. The location of each of the imagebubbles 131 may be stored in database 123 as an address or associationwith road segments. The image bubbles 131 may include graphicalrepresentation of nearby buildings 133 and the roadway 134.

FIG. 4 illustrates an exemplary enhanced composite image 151. Thecomposite image 151 may be stored as data representing a panoramicimage. The super-resolution panoramic image is built form multiple imagebubbles. For example, data from a first image bubble 152 is incorporatedwith data from a second image bubble 153. For example, data from thefirst image bubble 152 and data from the second image bubble 153 arecombined to form the composite image data. More image bubbles may beused. The super-resolution panoramic image may have a spatial resolutionthat is greater than the first image bubble 152 resolution and thesecond image bubble 153 resolution. Spatial resolution is the number ofindependent pixel values per unit length. Spatial resolution may bemeasured in units of pixels per meter. The spatial resolution for animage may vary within the image depending on how far object in the imagewere from the camera. The spatial resolution describes how much physicaldetail can be viewed per unit length or the size of the smallest objectthat can be seen in the image. In addition, the view angle of bubbles152 and 153 provide a better view angle of the building faces 154 and155. The ‘straight-on’ view angles also provide a higher and more evenlydistributed spatial resolution.

The original image bubble 151, the first image bubble 152 and the secondimage bubble 153 were captured from different positions and/or differentperspectives. In one embodiment, the data of the first image bubble 152and the data of the second image bubble 153 are combined to form anenhanced, composite version of image 151. The composite image 151 is acollage of multiple image bubbles. In another embodiment, the compositeimage 151 begins as an image derived from an image bubble, but the imagebubble is augmented. The augmented image bubble incorporates portions ofthe first image bubble 152 and/or the second image bubble 153. Theenhanced regions may be stored separately as individual regions that canbe requested on demand. Additionally, the enhanced regions may be joinedwith the original image 151 to create a new super-resolution image 151b. The enhancement regions may be computed in real-time or may bepre-computed and stored for later use.

In either embodiment, the image bubbles are captured from differentpositions and/or perspectives. The depth maps, which may have beencollected at the same times and locations as the image bubbles, are usedto align the subject matter of the image bubbles. Consider the image ofan object. The object is viewable from a current image bubble butrelatively low image detail is included for the object in the currentimage bubble. The object's face may also be at a shallow angle to thecamera and not clearly visible. A closer image bubble includes moredetailed data for the object and may view the object face from a betterview angle. Image data from the closer image bubble is projected onto athree-dimensional model. The three-dimensional model is the point cloudincluding depth values. The three-dimensional model may also be apreviously generated polygon model. The data that was projected onto thethree-dimensional model is transformed to the perspective of the currentimage bubble. Therefore, image data from the closer image bubble issuperimposed onto the current image bubble in the orientation of thecurrent image bubble such that in appears to increase the spatialresolution of the current image bubble. Far away objects in the currentimage bubble become clearer.

The server 123 may identify which portion of the current image bubble orthe composite image data 151 should be augmented or enhanced. In oneembodiment, all portions of the current image bubble or the compositeimage data 151 that include subject matter that is closer to anotherimage bubble is enhanced. In another embodiment, the user selects aportion of the image to be enhanced using a cursor or touch screen.

In another embodiment, the server 123 is configured to identify objectsof interest. The objects of interest may be significant buildings forroute features as defined from a map database. The objects of interestmay be any object with a flat surface greater than a threshold size. Thethreshold size may be measured in pixels (e.g., 100, 1000, or 10,000pixels) or by distance (e.g., 1 foot, 1 meter, or 10 meters). Flatsurfaces are identified using the depth values. The server 123 may beconfigured to analyze the image and compare image regions to thethreshold size.

Alternatively, the server 123 may perform an image processing algorithmon the image to identify objects of interest. The image processingalgorithm may incorporate one or more of edge detection, objectrecognition, optical character recognition, or feature extraction. Edgedetection identifies changes in brightness, which corresponds todiscontinuities in depth, materials, or surfaces in the image. Objectrecognition identifies an object in an image using a set of templatesfor possible objects. The template accounts for variations in the sameobject based on lighting, viewing direction, and/or size. Opticalcharacter recognition may identify one or more alphanumeric characterson the movable object. Feature extraction reduces an image into a set offeature vectors in order to identify an object in the image. Points ofinterest in augmented reality may be used.

As shown in FIG. 4, the building 154 and the building 155 have beenidentified as an object of interest. The composite image 151, which mayhave originated from an image bubble, may have already included lowerdetailed versions of the building 154 and the building 155. However,images bubble 152 and image bubble 153 are closer the building 154 andthe building 155, respectively, and include data with a greater spatialresolution. A depth map is used to rotate the image data out of theperspective of the images bubble 152 and image bubble 153 and/or resizethe data for alignment with the composite image 151. Thus, the compositeimage 151 becomes an enhanced resolution image.

FIG. 5 illustrates an exemplary alternate view selection. A geographicregion 160 includes a road, various buildings, and a sign 163. The sign163 is an object of interest. Image bubbles 161-162 are illustrated forreference and are not actually visible in the geographic region 160. Theview of a current image is shown by the geographic region 160. Thecurrent image may be a two-dimensional perspective extracted from animage bubble. This view may be from an interactive visual display frominside the image bubble. The edge of the sign 163 is visible in thecurrent image. However, the sign 163 may have a relatively low spatialresolution or the sign 163 may be at an angle that makes the sign 163not readable or otherwise blocked in the current image.

The server 123 is configured to receive a user or processor selectionfor an object of interest such as the sign 163. The selection maydescribe a point on the two-dimensional perspective. The server 163determines the three-dimensional location of the selected point byaccessing a depth map associated for a depth value. The server 123selects the optimal image bubble. The position and orientation of theimage bubble is used to compute its orientation relative to the selected3D point. This computation determines the precise section of the imagebubble that contains the view of the sign. A view of the object ofinterest is rendered from the optimal image bubble. The portion of thespherical image of the image bubble is projected onto a 2D image planeand delivered to the client.

Alternate algorithms to determine the optimal image bubble include adistance algorithm, an indexing algorithm, a surface normal algorithm,and a line of sight algorithm. Each of the example algorithms may beperformed by the server 125, the mobile device 122, or the workstation128. In each of the algorithms, a geographically referenced location isdetermined from the selection of the point. The set of image bubbles 131are identified based on the geographically referenced location.

In the distance algorithm, a distance of each of the set of imagebubbles 131 to the selected point is determined. The distance may bedetermined using two or three location components in the Cartesiancoordinate system. The distance may be calculated as the square root ofthe sum of the squared differences of each coordinate of the location ofthe image bubble and the location of the point. The distances arecompared, and the image bubble having the smallest distance to thethree-dimensional rendering is selected as the optimum image bubble.

In the indexing algorithm, a lookup table associates each of the set ofimage bubbles 131 with one or more addresses, geographic locations,address range, or geographic location range. The lookup table may bestored in database 123 or internally to the mobile device 122 or server125. The lookup table is queried with an address or the coordinates of ageographic location and returns the optimum image bubble.

The surface normal algorithm analyzes the direction of the selectedsurface plane and selects an image that best faces the surface. Forexample, if a billboard sign (perpendicular to the road) were selected,the closest image may only view the sign from the side of the billboard.In such a case, the billboard would not be readable from such a steepangle. Instead, the surface normal of the billboard selection is used tomake sure the closest image bubble is taken than also views thebillboard straight on instead of the front the side. The algorithm mayalso take into account the original viewpoint's view angle and location.

The line of sign algorithm utilizes depth information to determine theoptimum image bubble 181. The depth information, which may be LiDAR dataor another depthmap, describes a distance from the location where thedepth information was collected to an object at every angle. The server125, the mobile device 122, or the workstation 128 is configured tocompare the depth information to the geographically referenced locationdetermined from the selection of the point. If the depth information andthe geographically reference location are within a threshold of eachother, the image bubble has a line of sight to the selected point. Eachof the image bubbles with line of sight may be considered a preferredimage bubble. The server 125, the mobile device 122, or the workstation128 may subsequently apply the distance algorithm to select the closestimage bubble from among the preferred image bubbles with line of sight.Alternatively or additionally, the selected image bubble from thepreferred image bubbles is determined based on a desired perspective orangle of view. Alternatively, a polygon model may also be used todetermine line-of-sight.

FIG. 6 illustrates an exemplary display of the alternate view selectionof FIG. 5. The server 123 incorporates the view of the object into theoriginal two-dimensional perspective as a peek view 171. The peek view171 may be a picture-in-picture view of the object of interest. Adisplay 170 includes the combination of the original two-dimensionalperspective view of a portion of the current image bubble and an insetof the object of interest extracted from another image bubble.

FIG. 7 illustrates another exemplary alternate view selection. Apanoramic image 175 illustrates a geographic region include pedestrians,a roadway, and various buildings. The panoramic image 175 was collectedfrom a distance and angle such that detail in the area shown by circle176 has low detail or spatial resolution. Specifically, the road marking177 cannot be identified.

FIG. 8 illustrates an exemplary overhead view 190 of the area nearpanoramic image 175 of FIG. 7. Various image bubbles 131 were collectedat different locations along the roadway. A current image bubble 191 isthe image bubble that corresponds to the panoramic image 175. Aselection is received at screen location at 177 in FIG. 7 whichcorresponds to an area shown by circle 192. As illustrated by arrow 187,the selection corresponds to the 3D location area shown by circle 176.The server determines the 3D location of the selection using a depthmap.The image bubbles 131 are analyzed to determine which image bubble 131is closest to the selected point. The closest image bubble 178 ischosen, and image data, 177, is extracted from the closest image bubble178. As shown by the arrow 188, the extracted data is correlated withthe current image bubble 191 at a portion marked by circle indicated thealternate view selection 193. The extracted image is overlaid on theoriginal view, 191, as an inset image overlay and is placed at thelocation of the original pick point, 192.

FIG. 9 illustrates an exemplary display of the alternate view selectionof FIG. 7. The panoramic image 195, which may be panoramic image 175that has been enhanced, includes the alternate view selection 193including data extracted from the closest image bubble 178. Accordingly,the road marking 177 is clearly shown as left turn arrow. The images inthe alternate view selection 193.

FIG. 10 illustrates another alternative view display. The example ofFIG. 10 includes a top-down view 196 and a see-through model. Thesee-through model illustrates that alternate viewpoints may have a viewof objects not visible from the original perspective view. PerspectiveView A shows the original viewpoint. The building is obstructed bytrees. In this case, a two-dimensional region is defined around thebuilding which indicates which part of the scene is desired to always bevisible. This region may be defined by the user, or other map,navigation, or destination planning algorithm. An Alternate View bubbleB was chosen as the best view of the selected region around thebuilding. The alternative view display 197 illustrates that the twoviewpoints are combined to provide visibility of the building throughthe trees The alternative view display 197 displays the alternateviewpoint from the perspective of image bubble B. This inset imageprovides both a higher resolution image and a better view angle of thebuilding than the original View A.

In an alternate implementation, the alternative view display 198demonstrates that the alternate view is perspective corrected to make itappear as part of the original View A. This may be accomplished by usinga homography perspective transformation between the two given viewperspectives. The homography is used along with the image depth maps togenerate a perspective correct three-dimensional image transformation.In addition the depth maps provide a way to know which pixels shouldfrom View B should be mapped to View A. The depthmap from View B is usedto determine the three dimensional location of each color pixel in imageB. If the 3D location falls outside the selected region around thebuilding, the pixel is not included. Further, the depthmap from image Amay be used to compare the alternate view pixels to the correspondingpixels in view A. From View A, the distance of both pixels (from View Aand B) are compared. If pixel B is further than pixel A, then pixel B ishidden from View A. An algorithm uses this comparison to composite thetwo images together. For example, pixels that were original hidden maybe slightly dimmed to indicate they are ‘see-through’ pixels.

FIG. 11 illustrates an exemplary mobile device 122 of the navigationsystem of FIG. 2. The mobile device 122 may be referred to as anavigation device. The mobile device 122 includes a controller 200, amemory 201, an input device 203, a communication interface 205, positioncircuitry 207, a camera 209, and a display 211. The workstation 128 mayinclude at least a memory and processor and may be substituted for themobile device 122 in the following endpoint-based embodiments. In thefollowing endpoint-based embodiments, the mobile device 122 or theworkstation 128 performs a majority of the processing.

As an alternative to the example above in which the server 125 accessthe image bubbles and transmits only the image data necessary to themobile device 122, the memory 201 may be configured to store imagebubbles. The image bubbles stored at memory 201 may be a small set ofimage bubbles needed for the current location of the mobile device 122.The image bubbles are referenced to geographic locations. The geographiclocation of each image bubble may be the location from which the imagebubble was collected. The memory 201 may also store road segments andnodes, as discussed above, and store partial image bubbles or renderingsof geographic regions.

The input device 203 is configured to receive a selection of a point ona panoramic image of a geographic region. The input device 203 may beone or more buttons, keypad, keyboard, mouse, stylist pen, trackball,rocker switch, touch pad, voice recognition circuit, or other device orcomponent for inputting data to the mobile device 122. The input device203 and the display 211 may be combined as a touch screen, which may becapacitive or resistive. The display 211 may be a liquid crystal display(LCD) panel, light emitting diode (LED) screen, thin film transistorscreen, or another type of display.

The controller 200, which may be referred to as a processor, isconfigured to access a depth map from the memory 201 or anotherdatabase. The depth map is associated with a two-dimensionalperspective, such as a panoramic image, which may be a photograph. Thecontroller 200 is configured to choose a depth value from the depth mapthat corresponds to a point selected from the two-dimensionalperspective. The controller 200 is configured to select a portion of animage bubble based on the depth value. The selection may be based ondistance, line of sight, or an index. A portion of the selected imagebubble is incorporated into the two-dimensional perspective.

The incorporation of the portion of the image bubble into thetwo-dimensional perspective makes take many forms. In one example, theresulting image is a collage of multiple image bubbles. In anotherexample, the incorporation is a picture-in-picture or window that ispositioned near the selected point.

The size of the portion of the selection image bubble may bepredetermined. Alternatively, the size of the portion of the imagebubble may be selectable by the user. For example, the size may be setvia the input device 203 or another user interface. In addition, thepoint on the two-dimensional perspective or panoramic image is selectedfrom the input device 203.

The selection of the optimal image bubble may be made by directcorrelation in a table. For example, the memory 201 may store a tablewhere each possible point on the virtual rendering of the geographicregion is paired with or indexed by an image bubble. Alternatively, theselection of the optimal image bubble may be made via a selectionalgorithm. Examples selection algorithms are the distance algorithm, theindexing algorithm and the line of sight algorithm, discussed above,which may be performed by the controller 200.

The controller 200 may be configured to calculate a distance between ageographically referenced location of the selected point in thegeographic region and each of the set of image bubbles stored in memory201. The controller 200 may select the image bubble that is closest tothe selected point as the optimal image bubble. Alternatively, thecontroller 200 determines whether a line of sight exists between thegeographically referenced location of the selected point and a center ofeach of the plurality of image bubbles. The determination of whether aline of sight exists can be determined based on a depthmap generatedfrom a location near the center of the image bubble.

The positioning circuitry 207 is optional and may be excluded for themap-related functions. The positioning circuitry 207 may include aGlobal Positioning System (GPS), Global Navigation Satellite System(GLONASS), or a cellular or similar position sensor for providinglocation data. The positioning system may utilize GPS-type technology, adead reckoning-type system, cellular location, or combinations of theseor other systems. The positioning circuitry 207 may include suitablesensing devices that measure the traveling distance, speed, direction,and so on, of the mobile device 122. The positioning system may alsoinclude a receiver and correlation chip to obtain a GPS signal.Alternatively or additionally, the one or more detectors or sensors mayinclude an accelerometer built or embedded into or within the interiorof the mobile device 122. The accelerometer is operable to detect,recognize, or measure the rate of change of translational and/orrotational movement of the mobile device 122. The mobile device 122receives location data from the positioning system. The location dataindicates the location of the mobile device 122.

FIG. 12 illustrates an exemplary server 125 of the navigation system ofFIG. 2. The server 125 includes a processor 300, a communicationinterface 305, and a memory 301. The server 125 may be coupled to adatabase 123 and a workstation 310. The database 123 may be a geographicdatabase as discussed above. The workstation 310 may be used as an inputdevice for the server 125. In addition, the communication interface 305is an input device for the server 125. The communication interface 305receives data indicative of use inputs made via the workstation 128 orthe mobile device 122. In the following endpoint-based embodiments, theserver 125 performs a majority of the processing.

The memory 301 stores multiple panoramic images. The panoramic imagesmay have been collected at street level or from an aerial viewpoint. Thepanoramic images overlap such that objects in one panoramic image mayappear in one or more other panoramic images. The communicationinterface 305 is configured to transmit panoramic images to theworkstation 310 or the mobile device 122.

The memory 301 also stores depth maps. The processor 300 queries a depthmap using a point selected from a panoramic image. The horizontal andvertical locations on the panoramic image are two coordinates for thepoint. A depth value resulting from the query of the depth map providesa third coordinate. The third coordinate indicates how far the objectcorresponding to the selected point is from the location where the depthmap and the panoramic image were captured. The third coordinatealternatively may be referenced to another coordinate system (e.g., areference plane in the panoramic image).

The processor 300 determines an optimal image bubble for the point basedon the depth value according to any of the algorithms above. Theprocessor 300 incorporates at least a portion of the image bubble intothe panoramic image. The incorporation is an enhancement of thepanoramic image. The enhancement may increase the spatial resolution ofthe panoramic image in an area surrounding the object corresponding tothe selected point. The enhancement may augment the panoramic image withan insert or zoomed-in view of the object corresponding to the selectedpoint.

The processor 300 may calculate the rotation of the portion of the imagebubble from the perspective of the image bubble to the perspective ofthe panoramic image using homography. Homography is a geometric processincluding is an invertible transformation of a projective space. Ahomograph describes the arrangement of a point or a line from oneperspective to another. Using homography and the depth map, theprocessor 300 rotates and sizes the portion of the image bubble to alignwith the panoramic image.

The processor 300 may also manipulate the portion of the image bubblebased on user commands. The user commands may originate with the mobiledevice 122 or the workstation 310. A zoom commands may increase ordecrease the size of the portion of the image bubble. A rotate commandchanges the view angle of the incorporated image bubble by rotating theimage bubble about the center of the image bubble. The rotate commandmay rotate the incorporated image bubble away from the selected point.For example, a left or right arrow key or mouse movement pivots theimage bubble left and right, and an up or down arrow key or mousemovements pivots the image bubble up and down. In this way, the imagebubble is pivoted to show subject matter (e.g., buildings) at locationsother than the selected point.

A pivot command or a source image control mode causes lateral movementin the window. The processor 300 selected a subsequent image bubble todisplay in the window. The subsequent window may be automaticallyoriented to view the selected point. For each subsequent image bubble,the processor 300 is configured to generate an image including at leasta portion of the new image bubble in the window.

A perspective scroll command also switches between source images even inthe subsequent images do not view the selected point. A user or mobiledevice inputs a direction, and the processor 300 selects subsequentimage bubbles maintaining a centered or fixed view. The perspectivescroll may be achieved using a continuous image as described in moredetail by application Ser. No. 13/340,923, titled “Path Side Imagery”and filed Dec. 30, 2011, which is incorporated by reference in itsentirety.

The controller 200 and/or processor 300 may include a general processor,digital signal processor, an application specific integrated circuit(ASIC), field programmable gate array (FPGA), analog circuit, digitalcircuit, combinations thereof, or other now known or later developedprocessor. The controller 200 and/or processor 300 may be a singledevice or combinations of devices, such as associated with a network,distributed processing, or cloud computing.

The memory 201 and/or memory 301 may be a volatile memory or anon-volatile memory. The memory 201 and/or memory 301 may include one ormore of a read only memory (ROM), random access memory (RAM), a flashmemory, an electronic erasable program read only memory (EEPROM), orother type of memory. The memory 201 and/or memory 301 may be removablefrom the mobile device 100, such as a secure digital (SD) memory card.

The communication interface 205 and/or communication interface 305 mayinclude any operable connection. An operable connection may be one inwhich signals, physical communications, and/or logical communicationsmay be sent and/or received. An operable connection may include aphysical interface, an electrical interface, and/or a data interface.The communication interface 205 and/or communication interface 305provides for wireless and/or wired communications in any now known orlater developed format.

FIG. 13 illustrates an example flowchart for alternate viewpoint imageenhancement. The acts of the flowchart may be performed by anycombination of the server 125, the mobile device 122 and the workstation128, and the term controller may refer to the processor of any of thedevices. Additional, different, or fewer acts may be provided. Forexample, acts associated with the different modes may be provided.

An image is generated from an image bubble as part of navigation ormapping. Any selection or process may be used. Alternatively, an imageis to be generated.

At act S101, the controller receives data indicative of a point. Thepoint may be selected automatically by the controller based on subjectmatter surrounding the point. The subject matter may be identified as aflat surface, a building, a sign, or another specified object accordingto image processing techniques. The point may be selected via userinput. The user input may indicate a desired area for enhanced detail oralternative viewpoint. For example, the user may be requesting to read asign or other words in the distance or to see an objected that isobstructed in the current view. The selection of the point may be partof a navigation application. For example, the selected point maycorrespond to a destination or a point of interest along a route.

At act S105, the controller accesses a three-dimensional location. Thethree-dimensional is calculated from a depth map by the controller orahead of time. A depth value from the depth map corresponds to the pointon the two-dimensional perspective and allows the controller todetermine precisely where in three-dimensional space the point islocated. The depth map may be a three-dimensional model. The depth mapmay be collected using LiDAR or another optical distancing system.

The depthmap may also be used to determine the surface normal and/or thesize of the object. Therefore, the image bubble is selected using a 3Dpoint, surface normal, surface scale. The current view vector may alsobe used to assist the image selection.

At act S107, the three-dimensional spatial location of the point is usedto select an image bubble. The image bubble may be the closest to thespatial location or the closest to the spatial location without anobstruction. Alternatively, the image bubble may be indexed according tothe spatial location in a lookup table or have another relationship withthe point.

At act S109, the controller incorporates at least a portion of the imagebubble into the two-dimensional perspective. The portion of the imagebubble illustrates the object that includes the selected point. Thecontroller may build a composite image from the image bubble bycombining the portion of the image bubbles with other image bubbles. Theportion of the image bubble may be rotated to align with the perspectiveof the two-dimensional perspective.

Alternatively, the controller may generate an image including thetwo-dimensional view with a picture-in-picture including the portion ofthe image bubble. The selected point and surrounding area are shown witha greater spatial resolution than other portions of the two-dimensionalview. The controller may rotate or zoom the portion of the image bubbleaccording to user input. The controller may pivot or laterally scrollbetween adjacent image bubbles according to user input.

FIG. 14 illustrates an exemplary transition sequence for alternateviewpoint image enhancement. The alternate view display is used as atransition to move between image bubbles. A user navigates the currentimage bubble my exploring the scene of other viewpoints using thealternate view inset. When a desired alternate viewpoint is chosen, atransition effect takes the user to the alternate viewpoint and assignsthat image bubble as the new main viewpoint. A smooth transition isachieved by growing the alternate view inset window until it fills theentire screen as shown in FIG. 14. That is, the server 125 or mobiledevice 122 may be configured to receive a user command to switch imagebubbles and access a subsequent image bubble from memory based on thecommand.

This type of transition allows the user to move through multiple imagebubbles and multiple view angles seamlessly without the necessity of 3Dgeometry models. The transition is a type of ‘portal’ or teleportationin that the viewpoint moves to an alternate viewpoint location anddirection without the need to travel thought 3D space. Even though thedisplayed view is at a different 3D location and orientation, there isno need for the transition to perform any real 3D travel or 3D rotation.

The alternate view point transition may involve any number of successivetransitions between images and viewpoints. Thus the transition allows afree form image exploration of an entire scene. FIG. 15 illustrates twosuccessive transitions between three image bubbles. The initial View Alooks eastward down a road. The user selects an alternate view bypicking the sign and the view is displayed in an inset window as view B.The user may then further modify the alternate view by pivoting theviewpoint of the alternate view. In the illustration, the alternateviewpoint faces south, looking at the sign. The user requests to zoominto the new viewpoint and the transition to View B occurs. The maininterface now displays and manipulates the image bubble of View B. Notethat the viewpoint now faces south instead of the original eastwardview. The user selects another alternate view. This time by picking thefactory as seen from View B. The alternate View C is generated andshowing in the inset window. Next, the transition zooms to the View Cand this view becomes the main view and looks westward. The twotransitions resulted in moving the viewpoint location around a cornerand looking at the front of the factory building. A history oftransition may be stored either at the mobile device 122 or at theserver 125 and unwound to move back to the previous views B and A.

The embodiments above may be applied to or combined with on demandoverlays on three-dimensional renderings of buildings or other objectsas described by co-pending U.S. application Ser. No. 13/536,589 andpublished as US 2014/0002440, titled “On Demand Image Overlay” to JamesD. Lynch filed Jun. 28, 2012, which is incorporated by reference in itsentirety.

The network 127 may include wired networks, wireless networks, orcombinations thereof. The wireless network may be a cellular telephonenetwork, an 802.11, 802.16, 802.20, or WiMax network. Further, thenetwork 127 may be a public network, such as the Internet, a privatenetwork, such as an intranet, or combinations thereof, and may utilize avariety of networking protocols now available or later developedincluding, but not limited to TCP/IP based networking protocols.

While the non-transitory computer-readable medium is shown to be asingle medium, the term “computer-readable medium” includes a singlemedium or multiple media, such as a centralized or distributed database,and/or associated caches and servers that store one or more sets ofinstructions. The term “computer-readable medium” shall also include anymedium that is capable of storing, encoding or carrying a set ofinstructions for execution by a processor or that cause a computersystem to perform any one or more of the methods or operations disclosedherein.

In a particular non-limiting, exemplary embodiment, thecomputer-readable medium can include a solid-state memory such as amemory card or other package that houses one or more non-volatileread-only memories. Further, the computer-readable medium can be arandom access memory or other volatile re-writable memory. Additionally,the computer-readable medium can include a magneto-optical or opticalmedium, such as a disk or tapes or other storage device to capturecarrier wave signals such as a signal communicated over a transmissionmedium. A digital file attachment to an e-mail or other self-containedinformation archive or set of archives may be considered a distributionmedium that is a tangible storage medium. Accordingly, the disclosure isconsidered to include any one or more of a computer-readable medium or adistribution medium and other equivalents and successor media, in whichdata or instructions may be stored.

In an alternative embodiment, dedicated hardware implementations, suchas application specific integrated circuits, programmable logic arraysand other hardware devices, can be constructed to implement one or moreof the methods described herein. Applications that may include theapparatus and systems of various embodiments can broadly include avariety of electronic and computer systems. One or more embodimentsdescribed herein may implement functions using two or more specificinterconnected hardware modules or devices with related control and datasignals that can be communicated between and through the modules, or asportions of an application-specific integrated circuit. Accordingly, thepresent system encompasses software, firmware, and hardwareimplementations.

In accordance with various embodiments of the present disclosure, themethods described herein may be implemented by software programsexecutable by a computer system. Further, in an exemplary, non-limitedembodiment, implementations can include distributed processing,component/object distributed processing, and parallel processing.Alternatively, virtual computer system processing can be constructed toimplement one or more of the methods or functionality as describedherein.

Although the present specification describes components and functionsthat may be implemented in particular embodiments with reference toparticular standards and protocols, the invention is not limited to suchstandards and protocols. For example, standards for Internet and otherpacket switched network transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP,HTTPS) represent examples of the state of the art. Such standards areperiodically superseded by faster or more efficient equivalents havingessentially the same functions. Accordingly, replacement standards andprotocols having the same or similar functions as those disclosed hereinare considered equivalents thereof.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

As used in this application, the term ‘circuitry’ or ‘circuit’ refers toall of the following: (a) hardware-only circuit implementations (such asimplementations in only analog and/or digital circuitry) and (b) tocombinations of circuits and software (and/or firmware), such as (asapplicable): (i) to a combination of processor(s) or (ii) to portions ofprocessor(s)/software (including digital signal processor(s)), software,and memory(ies) that work together to cause an apparatus, such as amobile phone or server, to perform various functions) and (c) tocircuits, such as a microprocessor(s) or a portion of amicroprocessor(s), that require software or firmware for operation, evenif the software or firmware is not physically present.

This definition of ‘circuitry’ applies to all uses of this term in thisapplication, including in any claims. As a further example, as used inthis application, the term “circuitry” would also cover animplementation of merely a processor (or multiple processors) or portionof a processor and its (or their) accompanying software and/or firmware.The term “circuitry” would also cover, for example and if applicable tothe particular claim element, a baseband integrated circuit orapplications processor integrated circuit for a mobile phone or asimilar integrated circuit in server, a cellular network device, orother network device.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andanyone or more processors of any kind of digital computer. Generally, aprocessor receives instructions and data from a read only memory or arandom access memory or both. The essential elements of a computer are aprocessor for performing instructions and one or more memory devices forstoring instructions and data. Generally, a computer also includes, orbe operatively coupled to receive data from or transfer data to, orboth, one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio player, a Global Positioning System (GPS) receiver, to namejust a few. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a devicehaving a display, e.g., a CRT (cathode ray tube) or LCD (liquid crystaldisplay) monitor, for displaying information to the user and a keyboardand a pointing device, e.g., a mouse or a trackball, by which the usercan provide input to the computer. Other kinds of devices can be used toprovide for interaction with a user as well; for example, feedbackprovided to the user can be any form of sensory feedback, e.g., visualfeedback, auditory feedback, or tactile feedback; and input from theuser can be received in any form, including acoustic, speech, or tactileinput.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

The illustrations of the embodiments described herein are intended toprovide a general understanding of the structure of the variousembodiments. The illustrations are not intended to serve as a completedescription of all of the elements and features of apparatus and systemsthat utilize the structures or methods described herein. Many otherembodiments may be apparent to those of skill in the art upon reviewingthe disclosure. Other embodiments may be utilized and derived from thedisclosure, such that structural and logical substitutions and changesmay be made without departing from the scope of the disclosure.Additionally, the illustrations are merely representational and may notbe drawn to scale. Certain proportions within the illustrations may beexaggerated, while other proportions may be minimized. Accordingly, thedisclosure and the figures are to be regarded as illustrative ratherthan restrictive.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable sub-combination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a sub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings and describedherein in a particular order, this should not be understood as requiringthat such operations be performed in the particular order shown or insequential order, or that all illustrated operations be performed, toachieve desirable results. In certain circumstances, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components in the embodiments described above should notbe understood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

One or more embodiments of the disclosure may be referred to herein,individually and/or collectively, by the term “invention” merely forconvenience and without intending to voluntarily limit the scope of thisapplication to any particular invention or inventive concept. Moreover,although specific embodiments have been illustrated and describedherein, it should be appreciated that any subsequent arrangementdesigned to achieve the same or similar purpose may be substituted forthe specific embodiments shown. This disclosure is intended to cover anyand all subsequent adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, are apparent to those of skill in the artupon reviewing the description.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b) and is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, various features may begrouped together or described in a single embodiment for the purpose ofstreamlining the disclosure. This disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter may be directed toless than all of the features of any of the disclosed embodiments. Thus,the following claims are incorporated into the Detailed Description,with each claim standing on its own as defining separately claimedsubject matter.

It is intended that the foregoing detailed description be regarded asillustrative rather than limiting and that it is understood that thefollowing claims including all equivalents are intended to define thescope of the invention. The claims should not be read as limited to thedescribed order or elements unless stated to that effect. Therefore, allembodiments that come within the scope and spirit of the followingclaims and equivalents thereto are claimed as the invention.

I claim:
 1. A method comprising: receiving data indicative of a point ona two-dimensional perspective image, wherein the point is associatedwith a desired object; accessing, using a processor, a three-dimensionallocation computed from a depth map, wherein the depth map includes depthvalues for pixels of the two-dimensional perspective image; selecting,using the processor, an image bubble from a plurality of image bubblesbased on the three-dimensional location computed from the depth map,wherein the plurality of image bubbles are panoramic images; andincorporating, using the processor, a window inset including at least aportion of the image bubble with the two-dimensional perspective imagebased on the depth map, wherein the at least the portion of the imagebubble represents the desired object, wherein the at least the portionof the image bubble represents a different view angle than thetwo-dimensional perspective image.
 2. The method of claim 1, furthercomprising: receiving a command to manipulate the at least the portionof the image bubble by view angle or zoom.
 3. The method of claim 1,further comprising: receiving a command to switch to a subsequent imagebubble; and incorporating at least a portion of the subsequent imagebubble with the two-dimensional perspective image according to thecommand.
 4. The method of claim 1, wherein incorporating the at leastthe portion of the image bubble with the two-dimensional perspectiveimage comprises: generating a final image including the two-dimensionalperspective image and data derived from the at least the portion of theimage bubble.
 5. The method of claim 1, further comprising: rotating theat least the portion of the image bubble to approximate thetwo-dimensional perspective image.
 6. The method of claim 1, whereinincorporating the at least the portion of the image bubble with thetwo-dimensional perspective image comprises: projecting the at least theportion of the image bubble on a three-dimensional model; andtransforming data from the three-dimensional model to thetwo-dimensional perspective image.
 7. The method of claim 1, whereindetail from multiple image bubbles is incorporated with thetwo-dimensional perspective image.
 8. The method of claim 1, wherein afirst spatial resolution is associated with the desired object in thetwo-dimensional perspective image and a second spatial resolution isassociated with the desired object in the image bubble, and wherein thefirst spatial resolution is less than the second spatial resolution. 9.The method of claim 1, wherein the at least the portion of the imagebubble includes scenery obstructed in the two-dimensional perspectiveimage.
 10. An apparatus comprising: a memory storing a plurality ofimage bubbles; and a processor configured to access a three-dimensionallocation from a depth map associated with a two-dimensional image,select an image bubble from the plurality of image bubbles and select aportion of the selected image bubble, based on the three-dimensionallocation, to be incorporated, based on the depth map, with thetwo-dimensional image as a window inset, wherein the plurality of imagebubbles are panoramic images, wherein a first resolution associated withthe portion of the selected image bubble is greater than a secondresolution associated with the two-dimensional image at the pointselected from the two-dimensional image, wherein the portion of theselected image bubble at the first resolution and the two-dimensionalimage at the second resolution are displayed together at the same time,wherein the at least the portion of the image bubble represents adifferent view angle than the two-dimensional perspective image.
 11. Theapparatus of claim 10, wherein the point is selected from a userinterface and the two-dimensional image is a photograph.
 12. Theapparatus of claim 10, wherein the processor is configured to rotate andsize the portion of the image bubble to align with the two-dimensionalimage using the depth map.
 13. The apparatus of claim 10, wherein detailfrom multiple image bubbles is added to the two-dimensional image. 14.The apparatus of claim 10, wherein the portion of the image bubbleincludes objects obstructed in the two-dimensional image.
 15. Theapparatus of claim 10, wherein the processor is configured to change theportion of the image bubble by rotating the image bubble about a centerof the image bubble.
 16. The apparatus of claim 10, wherein theprocessor is configured to access a subsequent image bubble according toinput data and incorporate at least a portion of the subsequent imagebubble into the two-dimensional image.
 17. The apparatus of claim 16,wherein the portion of the subsequent image bubble includes an objectcorresponding to the point selected from the two-dimensional image. 18.A non-transitory computer readable medium including instructions thatwhen executed are operable to: determine a first image bubble associatedwith a first position and a first view angle, wherein the first imagebubble includes data indicative of an object at a first spatialresolution; identify a second image bubble associated with a secondposition and a second view angle in a region represented by the firstimage bubble, wherein the second image bubble includes data indicativeof the object at a second spatial resolution; generate an image with thefirst image bubble at the first view angle; and incorporate at least aportion of the second bubble into the image as a window inset andaccording to a depth map associated with the first image, wherein thefirst image bubble and the second image bubble are panoramic images. 19.The non-transitory computer readable medium of claim 18, the instructionoperable to: query the depth map using a point selected from a panoramicimage of the first image bubble to receive a depth value; determine thesecond image bubble as being for the point based on the depth value,wherein the at least the portion of the second image bubble isincorporated with the panoramic image.
 20. The non-transitory computerreadable medium of claim 19, wherein the second spatial resolution isgreater than the first spatial resolution.